A document conveying device including a rear surface reading unit which reads a rear surface of a document is known. Shading correction of the rear surface reading unit is performed by arranging a white reference member and the rear surface reading unit such that these face each other. The white reference member retreats from a position where the white reference member faces the rear surface reading unit when a document is conveyed. A drive source is needed to switch the position of the white reference member, but this would increase the number of drive sources and thus the manufacturing cost of the document conveying device. If a drive source in common with a paper ejection roller is used for moving the white reference member, there is a possibility that a document which is subjected to the ejection by the paper ejection roller is pulled back into a document conveying path when the white reference member rotates while the document to be ejected is still near the paper ejection roller.